Black-Box Data-efficient Policy Search for Robotics

The most data-efficient algorithms for reinforcement learning (RL) in robotics are based on uncertain dynamical models: after each episode, they first learn a dynamical model of the robot, then they use an optimization algorithm to find a policy that maximizes the expected return given the model and its uncertainties. It is often believed that this optimization can be tractable only if analytical, gradient-based algorithms are used; however, these algorithms require using specific families of reward functions and policies, which greatly limits the flexibility of the overall approach. In this paper, we introduce a novel model-based RL algorithm, called Black-DROPS (Black-box Data-efficient RObot Policy Search) that: (1) does not impose any constraint on the reward function or the policy (they are treated as black-boxes), (2) is as data-efficient as the state-of-the-art algorithm for data-efficient RL in robotics, and (3) is as fast (or faster) than analytical approaches when several cores are available. The key idea is to replace the gradient-based optimization algorithm with a parallel, black-box algorithm that takes into account the model uncertainties. We demonstrate the performance of our new algorithm on two standard control benchmark problems (in simulation) and a low-cost robotic manipulator (with a real robot).Comment: Accepted at the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2017; Code at http://github.com/resibots/blackdrops; Video at http://youtu.be/kTEyYiIFGP

Black-Box Data-efficient Policy Search for Robotics

International audienceThe most data-efficient algorithms for reinforcement learning (RL) in robotics are based on uncertain dynam-ical models: after each episode, they first learn a dynamical model of the robot, then they use an optimization algorithm to find a policy that maximizes the expected return given the model and its uncertainties. It is often believed that this optimization can be tractable only if analytical, gradient-based algorithms are used; however, these algorithms require using specific families of reward functions and policies, which greatly limits the flexibility of the overall approach. In this paper, we introduce a novel model-based RL algorithm, called Black-DROPS (Black-box Data-efficient RObot Policy Search) that: (1) does not impose any constraint on the reward function or the policy (they are treated as black-boxes), (2) is as data-efficient as the state-of-the-art algorithm for data-efficient RL in robotics, and (3) is as fast (or faster) than analytical approaches when several cores are available. The key idea is to replace the gradient-based optimization algorithm with a parallel, black-box algorithm that takes into account the model uncertainties. We demonstrate the performance of our new algorithm on two standard control benchmark problems (in simulation) and a low-cost robotic manipulator (with a real robot)

xOS: The End Of The Process-Thread Duo Reign

International audienceProcess and Thread are first-order abstractions of the operating system (OS), whose implementation is wired into the OS core. Several research works have shown the inadequacy of these two main abstractions for modern isolation needs, leading to the introduction of additional abstractions with new isolation and communication features. Despite their usefulness, these new proposals are introduced in a somewhat ad-hoc manner, compromising their broad and consensual adoption. This position paper presents xOS, an OS design that does not introduce yet another first-class isolation abstraction but instead investigates how the OS can help application programmers, libraries, and OS developers integrate and easily use new abstractions. To our knowledge, xOS is the first work in this area. Similar to file system development built around a Virtual File System (VFS), xOS introduces the concept of Isolation Context (IC), which should be the unique first-class abstraction wired into the OS core. ICs can be realized in several pluggable Isolation Context Factories (ICFs) such as ProcessFactory (provides processes), Thread-Factory (provides threads), Docker Engine (provides Docker containers), KVM (provides KVM virtual machines), Wasp (provides virtines), etc. We discuss our plan to redesign a general-purpose OS from these foundations, the required APIs, and how to support new and legacy applications